Insanity is delusional or psychotic. With respect to creativity, insanity creates things that are absurd, inauspicious or impractical.

Not that absurdity is always bad, perhaps Salvador Dali had some merit.

And of course, fiction doesn't have to be practical, I mean that if you're trying to create or engineer something in the real world, and it's incoherent or the principles involved are divorced from reality in an absurd way so that the creation is actually impractical in the real world, then that could be considered insane.

Regarding the term 'inauspicious', I mean that in art there's 100% leniency for creativity, anything goes, so what would separate something insane from something not? Well, beauty of course, but I think there are other dimensions too. You could have a vile, misaligned or malignant personality that creates things that are deleterious to the beholder in subtle ways. If you look at a piece of art, for example, you don't really know whether the artist is on your side or not. If he's insane, maybe the impression he's giving isn't really aiding you. Maybe it's misleading you.

I guess I shouldn't have mentioned vileness and malignancy, since the topic is specifically insanity. But insanity can also cause someone to have misaligned values or to see some aspect of the world incorrectly which could mislead the observer of the creative work.. or actually, what I was thinking maybe is that the insane person's art could be rabidly wild, exploring territory that shouldn't be explored, mixing things that shouldn't be mixed, stepping on things that shouldn't be stepped on, or being generally crass or like a bull in a china shop. Of course in our naivety, or at least in our grossly lacking understanding of the innards of human nature and the cosmos, art lovers may stand and appreciate this piece of art and try to glean understanding or experience from it, while having no idea of its deleterious nature on account of the artist being wildly or maybe dementedly insane.

It's not so much that there's a line between insanity and creativity--they can easily overlap. It's a question of whether the creation in question was made by a mad man or not, so it's like there's creativity and the subset of creativity that is also insanity. Of course, it can go the other way too--you can say there's insanity and then there's the subset of insanity that's creative. Personally I think I'd prefer the creations I come across not to be insane, but I'd prefer the insane people I come across to be creative.